专利摘要:
A system (12) for tracking and processing pen features, the system comprising a digital pen (1) for allowing a user to execute pen strokes on a substrate (2) and a receiving station (3) to use in combination with said digital pen (1) and adapted to follow said pen lines and to transfer said digital pen features which consist of electronic data representing said pen traces followed. Said receiving station (3) comprises a wireless transceiver adapted to communicate with a mobile terminal (4) via a wireless connection (9) for transferring said digital pen features to said mobile terminal (4).
公开号:BE1020233A3
申请号:E2011/0638
申请日:2011-10-31
公开日:2013-06-04
发明作者:Muelenaere Pierre De;Olivier Dupont;Patrick Verleysen;Michel Dauw
申请人:Iris Sa;
IPC主号:
专利说明:

System for tracking and processing pen lines handwritten on a mobile terminal
Technical area
The present invention relates to a system for tracking and processing pen features handwritten on a mobile terminal.
Prior art
Digital writing instruments (or styli), interchangeably referred to herein as "digital pens" whether or not they are written in ink, can be used to capture pen strokes and to digitize them. Digital writing systems comprising character recognition software can be used to convert recorded pen strokes into image type or text type data.
Typically, digital pens operate with a host terminal (such as a personal computer) and send data through a wired connection, and are added to the input with a keyboard or mouse. This can be done directly or indirectly via one or more receiving stations that receive a signal from the digital pen. The location of the digital pen is followed by a terminal tracking mechanism and the location data can be written to the memory of the host terminal.
On mobile devices with a touch screen such as smartphones, PDAs and tablet PCs, you can enter handwritten notes and sketches by touching the screen with your finger or stylus. Handwriting input directly on the touch screen has several limitations: ergonomic problems, very low accuracy, a screen much smaller than a typical paper notebook, and so on.
Many terminals do not offer the ability to receive pen strokes from a digital pen because they do not have an appropriate port for the wired connection. In the case of USB data transfer, even if the mobile computer and the digital pen have a USB port, neither can be configured as a host. In any case, the use of a wired connection between a mobile terminal and an external device is generally regarded as a major limitation to its usability.
Disclosure of the invention
An object of the invention is to provide a system which does not have at least one of the disadvantages mentioned above.
According to the present invention, the above-mentioned object is achieved with a handwritten pen stroke tracking and processing system, comprising: a digital pen for allowing a user to execute pen strokes on a substrate; a receiving station for use in combination with said digital pen and adapted to follow said pen stroke and to transfer said digital pen features which consist of electronic data representing said pen traces followed, said receiving station comprising a wireless transceiver capable of communicating with a mobile terminal via a wireless connection for transferring said digital pen features to said mobile terminal; and a software application executable on said mobile terminal and including software code fragments for receiving and processing said transferred digital pen strokes.
In embodiments according to the present invention, the processing may include one or more of the following: image processing, character recognition, image compression, and file format conversion into a text format or any combination format image and text, including data processed in an email, storage for further processing or otherwise. The subsequent processing may include anything above and may be performed on the mobile terminal (integrated processing) or externally, for example using a Software as a Service (SaaS) delivery model.
In embodiments according to the present invention, the wireless connection between the receiving station and the mobile terminal may be a Bluetooth® connection, a Wi-Fi connection or any other wireless connection. An RF wireless connection with a guaranteed bandwidth (or data rate) is preferably used for smooth transfer of the pen features.
In embodiments according to the present invention, the digital pen strokes may contain position information and a time stamp so that the order of the pen strokes can be reconstructed.
Brief Description of the Drawings The invention will be explained in more detail by way of the following description and the accompanying drawings.
Figure 1 shows a system for tracking and processing of pen lines handwritten according to the invention.
Figure 2 shows an example of combining a color or grayscale image with digitized pen features into a combined image.
Figure 3 shows an example of the combination of a color or grayscale image with the text corresponding to pen strokes converted to a combined image.
Figure 4 shows an example of the combination of a color or grayscale image with the pen stroke and with the text corresponding to the pen stroke converted into a combined image.
Figure 5 shows an embodiment of a receiving station for use in combination with a digital pen, and a way of attaching it to a substrate.
Figure 6 shows an example of the receiving station of FIG. 5 attached to the top of a page.
Figure 7 shows an example of the receiving station of FIG. 5 attached to the top right corner of a page.
Figure 8 shows several embodiments of the pen stroke processing according to the present invention as it can be implemented on the system of FIG. 1.
Modes of implementing the invention
The present invention will be described in connection with particular embodiments and with reference to certain drawings but the invention is however not limited thereto, being limited only by the claims.
The drawings described are only schematic and are non-limiting. In the drawings, the size of some elements may be exaggerated and not drawn to scale for illustrative purposes. The dimensions and the relative dimensions do not necessarily correspond to the actual functional embodiments (reductions to practice) of the invention.
In addition, the terms first, second, third and similar in the description and in the claims are used to distinguish between similar elements and not necessarily to describe a sequential or chronological order. The terms are interchangeable under the appropriate circumstances and the embodiments of the invention may operate in other sequences than those described or illustrated herein.
In addition, the terms up, down, above, below and similar in the description and claims are used for descriptive purposes and not necessarily to describe relative positions. The terms so used are interchangeable under appropriate circumstances and the embodiments described herein may operate in other orientations than those described or illustrated herein.
The term "comprising", as used in the claims, should not be construed as being limited to the means enumerated below; it does not exclude other elements or steps. It must be interpreted as specifying the presence of the elements, integers, steps or components referred to but does not exclude the presence or addition of one or more other elements, integers, steps or components or groups of these. Therefore, the scope of the phrase "a device comprising means A and B" should not be limited to devices consisting solely of components A and B. It means with respect to the invention that the only relevant components of the device are A and B.
Fig. 1 shows a system 12 for tracking and processing pen lines 13 on a mobile terminal 4, such as a laptop, netbook®, ipad®, tablet PC, PDA, or other suitable mobile devices. The pen strokes can be written by a hand-held writing instrument 1, for example a stylus, also referred to herein as "digital pen" 1. The digital pen 1 can write in ink on a substrate 2, preferably a writable writing surface such as paper, to also create an ink copy (also called "hard copy") of the writing or drawing that the user has made. The paper may for example be white paper with or without pre-printed lines or a square of rectangular grids. Other types of substrates may also be used, for example colored paper, printed paper, published paper, a printed form to be filled, but also for example a newspaper. .
The mobile terminal 4 is wirelessly connected to the receiving station 3 which is used in combination with the digital pen 1 and has software code fragments for capturing and recording the pen lines which are written by the user on the substrate 2 and can display the resulting graphic image on a display 5. The display is preferably integrated in the mobile terminal 4, for example the LCD screen of an ipad but can also be an external device. The screen 5 may or may not be a touch screen. The resolution of the screen 5 is not related to the resolution (or accuracy) of pen strokes entered. This means that handwritten pen lines can be entered in a much more detailed manner with much greater accuracy than is possible using the touch screen as an input device.
In one embodiment of the system 12, the mobile terminal 4 is provided with an application software for compressing the reconstructed graphic image 10 for archiving (for example on a hard disk or a flash card or other memory or a memory card). external network reader, etc.) or to share the image 10, for example by sending it on a network 6, for example by email.
In one embodiment of the system 12, the application software uses character (optical) recognition techniques to map the text type digital data 11 to the handwritten notes 13. The text type digital data 11 can be represented. for example as ASCII character codes (without any formatting information such as color, font type, font size, etc.) that can be saved as a text-only file ("text only"). Alternatively, the text type digital data 11 may also include formatting data such as, for example, color or font size or type of font, and may, for example, be saved as a document containing rich text (rich text for example an "RFT file") or as a Microsoft® Msword® document or as other document formats, for example WordPerfect®, MsWorks®, or other. But the text type digital data 11 can also be represented as unicode characters, which also allows the writing of languages such as Chinese or Japanese to be further processed.
The character recognition software that is used can convert the graphic data (such as, for example, a binary image (bitmap) 10 with a graphical representation of the handwritten notes 13) to text-type data 11 with position information and not to keep the attached drawings (graphical data 10). These may however contain important information that the writer has put in sketches. Since the character recognition software may make conversion errors (eg, character confusion), the digital image 10 of the handwritten notes 13 may be necessary in order to maintain a proper reference. Thus, it is preferred to use character recognition software that keeps both the converted text 11 and the (graphic) image 10 handwritten notes for archiving or e-mailing together.
In one embodiment of the system, illustrated in FIG. 4, the image 10 of the pen lines 13 is combined with the text 11 resulting from the character recognition software application, in a combined image 15 which keeps the graphic image 10, the text 11 and the coordinates of the elements of text. The combination can be a simple overlay (text or pen lines on the image) or a semi-transparent overlay or take place by other combination techniques such as, for example, color inversion. For example, the PDF and XPS formats are intended to include images, text and coordinates of text elements and corresponding viewing software applications (eg Acrobat® reader from Adobe®) can be used to display the image 15 and search in the text 11. When the search string is found, the corresponding graphic image part is retrieved and displayed on the screen (for example by automatically scrolling the document), the part of image corresponding to the search string that can be displayed differently, for example by highlighting it. This method requires a character recognition software application that produces the coordinates of the text elements (characters or words) in addition to the recognized text.
In one embodiment of the system, the image 10 of the handwritten notes 13 is combined with other images 7 such as, for example, the background image of a form to be filled, or an illustration (for example a photograph) or a screen shot or scanned image or other image. An example is shown in FIG. 2. As an example, the digital pen 1 can be used to fill out a paper form, the background image 7 of the form and the image 10 of the handwritten notes 13 are combined and saved together as a single file, to keep the correspondence between the two images. The background image 7 of the form may come from a file previously saved in the mobile terminal 4, which can be recalled and printed each time it has to be filled. Optionally, a digital photograph of the person completing the form can also be added to the combination of images. This photographic material may, for example, be previously stored in memory, or may, for example, be taken by a camera of the mobile terminal 4, for example a fixed camera or a video camera such as a camera. cyber camera (web-cam). The image resulting from the combination of the bitonal image 10 of pen lines with one or more other images 7 is a color or grayscale image (for example, the background of the form) with added text ( for example handwritten text as shown in Fig. 2 or reconstructed converted text as shown in Fig. 3). It is known that color or grayscale illustrations, for example digital photos, can be compressed using a compression algorithm such as, for example, JPEG or JPEG2000, which can offer a relatively high compression ratio (e.g. times or 10 times or more) while keeping a reasonable quality for most natural images. However, this is not the case for handwritten notes 13. In order to keep good readability of handwritten notes 13 and text 10 of the form (or other details of the image in color or in shades of gray), image can not be compressed at a high compression ratio, because the aforementioned methods (jpeg, jpeg2000, etc.) are mainly adapted for compressing continuous tone images such as digital photos, and typically produce artifacts highly visible when used on very detailed images like characters. This can be a problem because mobile terminals 4 generally have limited storage memory. For example, an ipadl® or ipad2® typically only has 16, 32 or 64 gigabytes of storage capacity, this space not only being used to store data but also being used to store the operating system, drivers and application software. Large files (for example, larger than 6 megabytes) can also cause problems when sending emails, not only because most email systems can only attach files that do not exceed a size limit ( for example 10 megabytes) but also because of the time required to transmit them.
Therefore, in one embodiment of the system of the present invention, the code of the basic software also includes code fragments for compressing the image resulting from the combination of the graphical image of the reconstructed pen traits and other images 7 using a high compression method which segments the image into bitonal data (pen lines, text, graphics) and color data (text and graphic colors, background colors, illustrations) and compresses these data separately therein with a compression method adapted to the type of data. The high compression method could follow the Mixed Raster Content (MRC) model, which is the subject of the ITU-T Recommendation T.44.
Figs. 5 - 7 show examples of a digital pen 1 and a corresponding receiving station 3 for tracking and recording the movements of the tip of the pen. Embodiments of the receiving station 3 used in the system 12 according to the present invention may be a recording device 3 to be attached which is attached to the substrate 2. FIG. 6 shows an example where the receiving station 3 is attached to the top of a paper. Fig. 7 shows an example where the receiving station 3 is attached to the upper right corner of a paper 2 but other places can also be used to position the receiving station 3. The digital pen 1 and the receiving station 3 can operate according to a mechanism ultrasonic pen tracking as described in US Pat. No. 7,839,394, "Electronic pen device" of Pegasus Technologies Ltd which is hereby incorporated by reference in its entirety. Alternatively, the pen tracking mechanism may also be another mechanism known to the skilled person such as, for example, followed by infrared laser or electronic tracking. Instead of an independent pen 1, a touch pad with a corresponding pen can be used in the system of the present invention.
The receiving station 3 may be a battery-powered device provided with an internal memory, (for example a volatile memory such as a DRAM memory or a non-volatile memory such as a flash memory), capable of working in autonomous mode (for example without being connected to the mobile terminal 4) to temporarily save followed pen strokes 13. The pen strokes are stored in the memory of the receiving station 3. The pen strokes can be downloaded later on the mobile terminal 4, saved or emailed as ordinary files. This system 12 allows the handwritten composition of e-mails. According to the invention, the pen lines are imported into the mobile terminal 4 using the wireless link 9. For this purpose, the software code fragments of the mobile terminal 4 incorporate the protocol stack of the wireless link technology. used.
A particularly interesting short-range wireless communication protocol is widely known as Bluetooth®. Bluetooth technology, also known as IEEE 802.15.1, operates in the 2.4 GHz license-exempt industrial, scientific, and medical (ISM) band and uses a broad-spectrum technique to minimize interference. The main specification (Core Specification V2.0 + EDR, released November 4, 2004) for Bluetooth is available at http://bluetooth.com. The content of this main specification is incorporated herein by reference in its entirety. Another interesting wireless communication protocol is Wifi, also known as IEEE 802.11 a / b / g / n. The mobile terminal 4 preferably has an integrated RF transceiver.
The receiving station and the mobile terminal can communicate using the Bluetooth protocol implemented from one side or the other by the protocol stack that manages the radio interface and processes the high level data.
The receiving station may include a radio that is arranged to communicate with the mobile terminal, appropriately via the Bluetooth wireless protocol. The radio is connected to a microprocessor arranged to control the transmission of signals between the receiving station and the mobile terminal.
The receiving station may include a silicon device containing the Bluetooth radio and a microprocessor that implements the protocol stack.
Mobile devices that support Bluetooth include a radio device and a protocol stack. Their protocol stack is implemented as part of their operating system and offers an API ("Application Programming Interface" - API for programming software) to the application software. The application software uses the API to check or request that Bluetooth be activated by the user, configures Bluetooth, finds the digital pen device, connects the digital pen device and manages the connection.
When the recorded pen stroke data is downloaded, the wireless link 9 is preferably configured such that the data is transferred at the highest speed in the shortest possible time while ensuring reliable transmission.
The receiving station 3 may also preferably be configured in a connected mode, or in some embodiments may only have the connected mode (i.e., connected to the mobile terminal 4). In this mode, the pen lines are continuously tracked and transferred as soon as possible to the mobile terminal 4 via the wireless link 9. In this mode, the wireless link 9 is preferably configured such that the delay between the pen and the mobile terminal 4 is reduced to a minimum. Alternatively, the wireless link 9 can also be configured in a low power mode, to save the battery life of the mobile terminal and / or the battery life of the receiving station 3.
In one embodiment, the pen 1 may, for example, be configured as a writing device, a force or pressure sensor built into the tip of the pen being used to detect when the tip of the pen is actually used to write (which pen strokes 13 are then recorded). Optionally, the pen 1 can also be configured to be used as a pointing device such as a mouse by which all pen movements are transferred to the mobile terminal 4 in real time. Optionally, Pen 1 can also have a push button to simulate a mouse click.
In one embodiment, the receiving station 3 stores in the memory a sequence of digitized positions of handwritten pen lines 13 and groups them as "virtual pages" corresponding to the physical papers on which it is written. The receiving station 3 preferably has means for changing the "virtual page", for example by using a push button, so that when the button is pushed, a new virtual page is started. The receiving station 3 may include enough memory to store, for example, 50 or 100 virtual pages or more, which may be ideal for students taking notes in class while having the ability to edit them later.
Fig. 8 illustrates some examples of software code fragments stored on the mobile terminal 4 of the system 12 according to the present invention. In the software code fragment 101, the mobile terminal 4 receives digital pen features 13 as digital data. In the software code fragment 102, this digital data is converted into a graphic image 10. In the optional software code fragment 103, this image file 10 (for example a bitonal image) can be displayed on the screen 5 of the software. mobile terminal 4, it can be edited, copied, etc., and / or saved as a graphic image file 10. The system 12 also preferably includes software code fragments 104 for converting the graphic image 10 including the digitally representing the pen strokes 13 into text, eg ASCII text or Unicode text as described above, performing character recognition. Optionally, the system may also include software code fragments 105 for editing, spelling, and / or saving the resulting text file, such as plain text (TXT file format) or rich text (file format) RTF) or other desired formats. It should be noted that a search can be performed in these text files, that is to say that a given character string can be easily and reliably found in such a text file. Optionally, the software code fragments 105 also include fragments for translating the digital text 11 into another language (for example, the writer may write notes on page 2 in Dutch language and the digital text may be translated into English by an automatic translation system). As a non-limiting example, the software module 105 can use "google translate" (freely available at the time of writing to the URL: "http://www.google.be/language_tools hl=nl"). Optionally, the system of the present invention also includes software code fragments 108 for converting digital text 11 (or translated digital text) to synthetic voice using a text-to-speech converter. These converters can use commercially available modules from multiple software vendors (eg Nuance Communications®). The resulting voice may be stored in a voice file 19, for example in a WAV or MP3 format, or in any other appropriate format. The system preferably also includes software code fragments 106 for combining the pen stroke information and / or the corresponding digital text information with other grayscale or color background images. as described above. The combined file 15 is preferably compressed so that the digitized pen lines 13 are kept at their highest recorded resolution, while the remainder of the image (mainly belonging to the background image 7). can be compressed at a higher rate. For this purpose, the system of the present invention preferably includes software code fragments 107 for compressing such a combined file. A hyper-compression algorithm is preferably used. Examples of such a high compression method are disclosed, for example, in US Patent Applications US 5778092 (A) and US 2008273807 (A1) both of which are incorporated herein by reference in their entirety. Both algorithms use a low-resolution foreground plane and a low-resolution background plane and a high-resolution bit plane to achieve the required compression and a high resolution for text 11 or the pen features scanned. The system 12 of the present invention preferably also includes software code fragments 109 for sending any combination of the image file 10, the text file 20, the voice file 19 or the compressed combined file 17 via a email.
The mobile terminal 4 can operate with Microsoft® Windows® operating system Microsoft® standard such as Windows XP® or Microsoft® Windows 7®, but other operating systems can also be used as, for example , iOS, Android, Blackberry OS, Windows Phone 7, HP webOS, or whatever.
The software application code fragments mentioned above may be independent programs or may be embedded or integrated into pilot software or may be provided as plug-ins for cooperating with existing software applications such as, for example, Powerpoint® which is part of of Microsoft® Office®, or Evernote® from Evernote Corporation but may also be provided in other ways known to those skilled in the art.
权利要求:
Claims (12)
[1]
A system (12) for tracking and processing pen features, the system comprising a digital pen (1) for allowing a user to execute pen features on a substrate (2) and a receiving station (3). ) to be used in combination with said digital pen (1) and able to follow said pen lines and to transfer said digital pen lines which consist of electronic data representing said pen traces followed, characterized in that said receiving station (3) comprises a wireless transceiver adapted to communicate with a mobile terminal (4) via a wireless connection (9) for transferring said digital pen features to said mobile terminal (4); and in that the system further comprises a software application executable on said mobile terminal (4) and comprising software code fragments for receiving said digital pen features transferred through said wireless connection (9) and to process said transferred digital pen features.
[2]
The system (12) of claim 1, wherein said digital pen strokes include position information and a time stamp so that the order of the pen strokes can be reconstructed.
[3]
The system (12) of claim 1 or 2, wherein the wireless connection between the receiving station and the mobile terminal is an RF wireless connection with a guaranteed bandwidth or data rate.
[4]
4. System (12) according to any one of the preceding claims, wherein the wireless connection between the receiving station and the mobile terminal is one of the following connections: Bluetooth® connection or Wifi connection.
[5]
The system (12) according to any one of the preceding claims, wherein the software application comprises software code fragments for detecting from among said digital pen strokes those which represent handwritten characters.
[6]
The system (12) of claim 5 wherein said software application includes software code fragments for performing handwriting recognition to convert digital pen strokes that represent handwritten characters into numeric characters.
[7]
The system (12) according to any one of the preceding claims, wherein the software application comprises software code fragments for transforming said digital pen features into graphic images that can be displayed on a display (5) of said terminal mobile (4) and graphically represent the pen stroke executed by the user.
[8]
The system (12) of any one of the preceding claims, wherein the software application comprises software code fragments for converting a graphic image containing a combination of a plurality of said digital pen strokes (7) and other color images in a hyper compressed document (16) in which different elements of the graphic image (10) are compressed with different compression algorithms optimized for the respective elements.
[9]
The system (12) of claim 8, wherein said software code fragments for said conversion are further provided to generate a file in which an electronic search can be performed and containing the hyper compressed document and recognized text starting of said plurality of digital pen strokes.
[10]
The system (12) of any one of the preceding claims, wherein the software application comprises software code fragments for embedding an image or text generated from said digital pen traits (7) into an electronic mail (18).
[11]
The system (12) of any of the preceding claims, wherein the software application comprises software code fragments for temporarily storing the digital pen features or the electronic data generated therefrom for further processing. at a later stage.
[12]
The system (12) of claim 11, wherein the system further comprises an external processing device adapted to communicate with said software application on said mobile terminal and to perform said further processing on said temporarily stored digital pen features or electronic data generated therefrom.
类似技术:
公开号 | 公开日 | 专利标题
US20120299881A1|2012-11-29|System for tracking and processing handwritten pen strokes on mobile terminal
JP5846130B2|2016-01-20|Position bar and bookmark function
US10778928B2|2020-09-15|Device and method for inputting note information into image of photographed object
US8244074B2|2012-08-14|Electronic acquisition of a hand formed expression and a context of the expression
JP5884741B2|2016-03-15|Automatic attachment of captured images to documents based on context
KR101037240B1|2011-05-25|Universal computing device
US9530050B1|2016-12-27|Document annotation sharing
US20120287070A1|2012-11-15|Method and apparatus for notification of input environment
US20160259766A1|2016-09-08|Ink experience for images
KR101366252B1|2014-02-20|Method and apparatus for printing data of predetermined format using bluetooth communication, method for storing template data
EP3291100B1|2019-04-10|Method and system for input areas in documents for handwriting devices
US20140145974A1|2014-05-29|Image processing apparatus, image processing method and storage medium
US20100045785A1|2010-02-25|Note Capture Device
US20080229186A1|2008-09-18|Persisting digital ink annotations as image metadata
US20220050975A1|2022-02-17|Content Translation Method and Terminal
BE1020233A3|2013-06-04|SYSTEM FOR MONITORING AND PROCESSING MANUSCRIPT PEN STRIPS ON A MOBILE TERMINAL.
US8718374B2|2014-05-06|Method and apparatus for accessing an electronic resource based upon a hand-drawn indicator
JP2009105897A|2009-05-14|Data management apparatus and data management method
JP5459046B2|2014-04-02|Information processing apparatus, information processing method, program, and information processing system
US20080282138A1|2008-11-13|Methods and systems for multimedia object association to digital paper spatial diagrams
JP6561547B2|2019-08-21|Computer program and image processing apparatus
JP2012194837A|2012-10-11|Image processing device, method, program, and recording medium
KR100974539B1|2010-08-11|Installtion for management of dictionary information in electric device and method thereof
KR20130080745A|2013-07-15|Method of generating electronic documents using camera module of smart phones and ocr engine of remote server, and terminal device using the same
US20210303779A1|2021-09-30|Sharing of user markings between printed and digital documents
同族专利:
公开号 | 公开日
引用文献:
公开号 | 申请日 | 公开日 | 申请人 | 专利标题
WO2008054160A1|2006-11-01|2008-05-08|Yong-Jik Lee|Messenger system for transmitting handwriting instant messages through wire and/or wireless network system and messenger service method thereof|
法律状态:
优先权:
申请号 | 申请日 | 专利标题
US201113490675A| true| 2011-05-27|2011-05-27|
US201113490675|2011-05-27|US13/462,122| US20120299881A1|2011-05-27|2012-05-02|System for tracking and processing handwritten pen strokes on mobile terminal|
[返回顶部]